Towards a Database of High-dimensional Plenoptic Images

The plenoptic function was introduced as a convenient, ray-based model for light that includes the color spectrum as well as spatial, temporal, and directional variation [1]. Throughout the last decade, an ever increasing number of approaches to acquiring individual plenoptic dimensions have been proposed [4]. We discuss the acquisition of experimental datasets that capture a variety of different combinations of plenoptic dimensions. We make these datasets publicly available and hope to advance the study of redundancies within the plenoptic function of natural scenes [2]. The provided data is also useful for simulating new approaches to computational plenoptic imaging as well as comparing alternative acquisition approaches. Dataset Acquisition A major challenge for the acquisition of high-dimensional plenoptic datasets are the limited computational resources available today. Imaging a scene that is photographed with an image resolution of 1024×768 pixels, 30 spectral bands, and from 15 × 15 slightly different viewpoints. The required storage space for a single image, excluding video, is about 160GB if each pixel is encoded in 32 bits to account for a high dynamic range. The sheer amount of data makes any on-board camera or otherwise on-line processing impossible with today’s computational power. We captured two different kinds of datasets that encompass different combinations of plenoptic dimensions: multi-spectral light fields and multi-spectral video (see Figs. 2, 3, and 4). The light fields are acquired by mounting a custom multi-spectral camera on a programmable X Y translation stage (Fig. 1). The camera consists of collimating optics, a liquid crystal tunable filter (LCTF), and a USB machine vision camera. For each viewpoint of the light field, we move the camera on the translation stage and capture an exposure sequence for a number of pre-defined narrow-band color channels. High dynamic range images are essential for precise capture of contrast in the scene and also to account for the wavelength-dependent attenuation of the LCTF. In a pre-processing step, we photograph a calibration grid from each camera position at a single wavelength to determine the extrinsic parameters and rectify all captured images. A one-time intrinsic camera calibration is necessary as well. Figure 1. A custom multi-spectral camera (right) mounted on a gantry (left) allows us to acquire multi-spectral light fields. For the acquisition of multi-spectral datasets that include scene motion, we position objects on programmable translation or rotation stages. These objects are moved between successive frames of a sequence, where each spectral image is assembled from multiple exposures. Discussion As recently demonstrated, multiplexing the dimensions of the plenoptic function on a sensor can be modeled as a unified image formation and, more importantly, reconstruction framework [3]. So far, the correlations between plenoptic dimensions have almost exclusively been exploited for color demosaicing. Here, highresolution tri-chromatic images are computed from RAW sensor images that sample different color channels on a spatially interleaved grid. Our datasets provide a first step toward the development of natural image statistics that encompass all dimensions of the plenoptic function and their correlations.